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Abstract 

The potential for associative recall of diluted neuronal networks is investigated with respect to 
several biologically relevant configurations, more specifically the position of the cells along the 
input space and the spatial distribution of their connections. First we put the asymmetric Hopfield 
model onto a scale-free Barabasi- Albert network. Then, a geometrical diluted architecture, which 
maps from L-bit input patterns into iV-neurons networks, with R = N/L < 1 (we adopt R=0. 1, 
0.2 and 0.3), is considered. The distribution of the connections between cells along the one- 
dimensional input space follows a normal distribution centered at each cell, in the sense that cells 
that are closer to each other have increased probability to interconnect. The models also explicitly 
consider the placement of the neuronal cells along the input space in such a way that denser regions 
of that space tend to become denser, therefore implementing a special case of the Barabasi-Albert 
connecting scheme. The obtained results indicate that, for the case of the considered stimuli and 
noise, the network performance increases with the spatial uniformity of cell distribution. 

1 Introduction 

A number of mathematical concepts and tools have been extensively applied in order to construct 
models and simulations of neuronal behavior that could shine some light into the intricate work- 
ings of the brain. In physics, particular importance and interest have been placed on Hopfield 
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models because of their potential for associative recall and close relationship with statistical me- 
chanics 12. However, while most such approaches have considered fully connected structures, 
experimental evidences indicate that biological neuronal networks involve a combination of in- 
tense local interconnections (responsible, among other things, for lateral inhibition \2\) coexisting 
with long range mappings which often exhibit topographical structure, in the sense that the con- 
nections preserve the spatial adjacency of the signal. Studies addressing partially connected Hop- 
field networks on lattices have appeared sporadically in the literature [3]. More recently [4|, the 
Barabasi-Albert [5 1 connecting scheme, where the "rich gets richer", was used to define Hopfield 
architectures with symmetric couplings whose potential for associative recall was assessed with 
respect to the number of connections between the constituent cells. The obtained results indicated 
that, despite the relatively sparse and heterogeneous connections, for suitable parameters such 
architectures preserved a good deal of the associative recall when compared to fully connected 
counterparts, bridging the gap to further investigations. 

The present work reports the extension of the models described in [ 6 , 4 1 in three aspects that 
are specially relevant from the biological point of view, namely the asymmetry that neuron i may 
influence neuron k even if neuron k has no influence on neuron i, the spatial distribution of cells 
along the input space (assumed to be a one-dimensional vector), and the spatial distribution of the 
connections between those cells. More especifically, as the network is constructed, each added 
cell is placed preferentially at regions of the input space that are denser in cells, in such a way 
that dense regions tend to become denser. Being diluted, not every bit of the input pattern will be 
connected to a neuron. Also, the connections between a cell i and the rest of the network are estab- 
lished in such a way that cells that are closer to each other have higher probability to interconnect, 
which is achieved through Monte Carlo simulation assuming normal weighting centered at cell 
i. The potential of the networks for associative recall is experimentally assessed with respect to 
several parameter configurations, with special attention given to the effect of the uniformity of the 
cell spatial distribution over the obtained performance. 

The article starts by revising some key aspects of biological neuronal structures and proceeds 
by presenting the obtained results and respective discussion, with special attention placed on 
the biological interpretation and implication of the observed behavior in terms of the parameter 
configurations. 

2 Spatial Organization in Biological and Computational 
Networks 

First, in what we call the asymmetric Hopfield-Barabasi- Albert model, we put the Hopfield model 
onto the scale-free Barabasi-Albert [5| network. In contrast to spin glasses and many physics 
systems, the interactions between biological neurons are not symmetric: Neuron k may influence 
neuron i even if neuron i has no influence on neuron k. In other words, the synaptic strengths 
J(i, k) are not symmetric, i.e. J(i, k) is not equal to J(k, i). We thus construct our model in such 
a way that every neuron i selects itself plus exactly m other neurons (repeated connections are 
allowed) as neighbours. This fixed number m + 1 of influencing neighbours greatly simplifies the 
algorithm. 

Second, in what we call the geometrical model, we try to incorporate the geometry instead 
of only the topology of neural nets. Another characteristic feature exhibited by the mammalian 
cortex is the spatial structuring permeating both the input signal and neuronal architectures. For 
instance, the visual field is mapped onto the retina space, which innervates, through the lateral 



2 



geniculate nucleus, into the primary visual cortex in such a way that the spatial adjancency of 
the input patterns is maintained. The importance of such topographical mappings is corroborated 
by the fact that they are found ubiquitously along the cortex, and not only in its primary sensory 
regions. It should be observed that such mappings are not isometric. One important neuronal 
pheonomenon closely related to topographical mappings is known as lateral inhibition, where the 
activity at a given cell is inhibited by its spatial neighbours along the input space. In spite of its 
key role in cortical organization, relatively few neuronal networks have explicitly taken spatial 
organization into account, Kohonen's SOM model [7] being one of the pioneering exceptions. At 
the same time, much of the efforts devoted to Hopfield models have systematically overlooked the 
spatial structure of input and topographical mappings. The situation is similar in the biological 
community, where only more recently (e.g. [8 1) attention was focused on this important organiza- 
tional principle. The fact that neuronal cells that are closer together tend to have increased chance 
to interconnect (e.g. [2]) has also been often overlooked. Indeed, cortical organization seems 
to combine local, intensive connections with long distance, sparser, mappings between distinct 
neuronal modules. Such local/global organization may also underly other spatial scales structures 
in the brain, possibly leading to scale-free characteristics over limited spatial scale intervals. In 
the current work, we propose a novel variation of the Hopfield model for associative recall that 
explicitly takes into account the spatial organization of the neuronal cells and the localized and 
global spatial distribution of their connections. 

The input space is assumed to be the one-dimensional vector i = 1, 2, . . . , L, to which 
a total of N neuronal cells are attached. The number of connections made by each cell is fixed 
and equal to m. The network is diluted, in the sense that R = N/L < 1. Figure 1 illustrates a 
typical realization of the assumed neuronal architecture for N = 7, m = 3 and L = 15, with the 
respective asymmetric weight matrix shown in Equation (Q. Each line i shows the connections 
of neuron i to other neurons by a ' 1 ' . The construction of the network involves the two following 
steps: (a) neurons are placed at spatial positions p G 0, 1, . . . , L according to a probability density 
function h(p) (cells can only be added at empty sites); and (b) m not necessarily distinct synaptical 
connections are established between the just added cell and its neighbours according to a normal 
(Gaussian) density function g a (d), where d is the distance between the position p of the reference 
cell and each of the other cells in the network and a is the standard deviation of the normal 
distribution. It is assumed throughout this article that L = 1000. 
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Figure 1: Example of the networks considered in this article assuming iV = 7, m = 3 and L = 15. 
The black squares stand for the position of the neurons. 
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A non-stationary (along time) density function h(p) is obtained at each interaction during the 
network construction by adding the function f(i) = 1 if \i—p\ < a and otherwise to the previous 
instance of h(p), where p is the position of the most recently added cell, and renormalizing h{p). 
Standard Monte Carlo sampling is then performed over the respective distribution function so as to 
select the position p for a new cell. The probability function h(p) is initially set as uniform. Monte 
Carlo simulation is also employed to select m cells for symmetric connections with the current 
cell. The connections are performed as follows: (i) a vector v(i) is built such that v(i) = 1 at the 
positions i where neurons exist and otherwise; (ii) this vector is weighted according to a normal 
density function centered at p, the position of the current cell; and (iii) Monte Carlo is used to 
select m sites of v. It is observed that all cells are always self-connected. Figure 2 illustrates 
typically obtained connection matrices. The reinforced diagonal structure of such matrices favors 
conditioning and reflects the locality of the synapses at specific spatial scales defined by a. 

For both the Hopfield-Barabasi- Albert and the geometrical model, the neurons i, i = 1,2, . . . N, 
in our models are either firing (Si = 1) or silent (Si = — 1) and are updated according to 



with synaptic strengths = Yin £f (Hebb rule) if i and k are connected. Here = ±1, fj, = 
1, 2, ... P, are P random bit-strings called patterns, and one of them is supposed to be recalled by 
this updating rule if it is presented to it in a perturbed fashion: Associative memory. The quality 
of recall is measured by the overlap ^ = J2i SiCl without loss of generality, the first pattern is 
supposed to be recovered. 



For the asymmetric Hopfield-Barabasi-Albert model, N binary neurons are added to the initial 
core of m + 1 neurons, and P binary patterns are stored. Initially, the first of these patterns 
is presented to the network with ten percent of the sites flipped (overlap = 0.8), and after a few 
iterations the dynamics comes to a fixed point for which the overlap ^ is determined. Fig. 3 shows 
as in the symmetric case that for P < m < iV the desired pattern is fully restored. However, 
Fig. 4 shows that the scaling law: ^ = f(m/P) for infinite N is not fulfilled well, in contrast to 
the symmetric case. The distribution of the number of neurons, influencing q sites each, follows 
roughly a 1/q 3 law for small q but not for large q (not shown). 

For the geometrical model, simulations were performed considering combinations of m = 25 
and 50 with a = 10 and 100. Parallel updating is adopted for Hopfield recovery and the number of 
trained patterns is P = 0.05iV. Relative overlaps are used , i.e. R = ty/N. The relative overlaps 
are calculated taking into account only the neurons attached to the input pattern. 

Figure 5 shows the correlation integral C(r) [9], namely the number of neurons with distance 
along the input space smaller or equal to r divided by A^ 2 . Figures 5 to 7 show examples of 



Si -> sign(^] JikSk) 



k 
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Figure 2: Example of connection matrices, shown as gray-color images, for m = 25 and a = 10 (a) 
and 100 (c), and for m = 50 and a = 10 (b) and 100 (d). Black means unconnected neurons, white 
means connected neurons. 



typically obtained density and respective distribution functions obtained for a = 10 (a) and 100 
(b) and considering 10 different realizations of the spatial distribution of cells for each parameter 
settings. 



4 Discussion 

For the Hopfield-Barabasi- Albert model, the asymmetric version gave results similar to the sym- 
metric version [4|, provided self-interaction in taken into account in both cases. For the geomet- 
rical model, as indicated by Fig. 5, the considered scheme for positioning the neurons along the 
input space led to similar power law behavior for a = 10 and 100 for the considered correlation 
lags. At the same time, as is clear from Fig. 6, the value of a strongly influences the charac- 
teristic spatial distribution of neurons, in the sense that shorter clusters of cells, indicated by the 
clumps in Fig. 6, are obtained for smaller values of a. As expected, higher values of m tend to 
produce more widespread distribution of synaptic connections, leading to the wider dispersions 
around the main diagonal of the matrices in Fig. 2(b) and (d) and substantially superior potential 



5 



Unnormalized overlap, 10000 neurons outside core of m neurons; ten samples, P = 10 (+) and 100 (x) 
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Figure 3: ^ for 10000 = m neurons (m in the core, surrounded by 10,000 others), for 10 (left) and 
100 (right) patterns. The curve gives the overlap for full recovery. 



for associative recall, shown in Fig. 7. This is a consequence of the fact that a larger number 
of connections has direct impact over the amount of memory in the network. At the same time, 
the network performance tends to improve as a is increased (see Fig. 7). A more uniform spatial 
distribution of the neurons is obtained for larger values of a, with the limit case a — > oo tending 
to the uniform distribution. It is clear that, at least for the considered kind of stimuli and noise 
(uniform distribution of signal changes), the effect of irregular distribution of neurons along the 
input pattern space tends to deteriorate associative recall. The important biological implication 
is that the development (ontogeny) of real neuronal systems coping with uniformly distributed 
stimuli and noise has to incorporate mechanisms for ensuring uniform distribution of neuronal 
placement. In addition to topographical maps, other possible biological mechanisms that could be 
involved in such spatial tuning may involve controled apoptosis (i.e. programmed neuronal cell 
death) and chemotatic migration modulated by density. It is important to bear in mind that dif- 
ferent results could be obtained for distinct input stimuli and noise models. For instance, stimuli 
characterized by specific correlation lengths could be better processed by networks with neurons 
and connections presenting congruent spatial characteristis. 

All in all, the present work has considered extensions of the classical Hopfield network which 
are characterized by enhanced biological plausibility as far as asymetric connections and spatial 
distribution of cell bodies and synapses are concerned. Such models were analysed according to 
the shape-function paradigm, i.e. by assessing how the model parameters influenced the asso- 
ciative recal performance, and a series of perspectives has been opened for further investigation. 
A particularly interesting issue being currently addressed regards the characterization of the ef- 
fect of the individual morphology of real neuronal cells over the network behavior, as well as the 
influence of stimuli and noise spatial features over the respective network performance. 



6 




Normalized overlap from 10 samples, N+m = 10000 neurons, P = 10 (+) and 100 (x); m=2 to 2000 
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Figure 4: \I//10000 for m neurons in the core and N = 10000 — m neurons surrounding them, for 
100 and 1000 patterns. If scaling would be valid the two data sets would overlap in this plot. 
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Figure 5: The loglog graph of the correlation integral C(r) obtained for a = 10 (a) and 100 (b). 



Figure 6: Typically obtained density (up) and respective distribution functions (down) obtained for 
a = 10 and 100 (b). 



Figure 7: Relative overlaps for m = 25 and a = 10 and 100 (upper pair of curves) and m = 50 and 
a = 10 and 100 (lower pair). 
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